Systems and methods for implementing time delay integration imaging techniques in conjunction with distinct imaging regions on a monolithic charge-coupled device image sensor

ABSTRACT

Systems and methods in accordance with embodiments of the invention implement TDI imaging techniques in conjunction with monolithic CCD image sensors having multiple distinct imaging regions, where TDI imaging techniques can be separately implemented with respect to each distinct imaging region. In many embodiments, the distinct imaging regions are defined by color filters or color filter patterns (e.g. a Bayer filter pattern); and data from the distinct imaging regions can be read out concurrently (or else sequentially and/or nearly concurrently). A camera system can include: a CCD image sensor including a plurality of pixels that define at least two distinct imaging regions, where pixels within each imaging region operate in unison to image a scene differently than at least one other distinct imaging region. In addition, the camera system is operable in a time-delay integration mode whereby time delay-integration imaging techniques are imposed with respect to each distinct imaging region.

CROSS-REFERENCE TO RELATED APPLICATIONS

The current application is a continuation of U.S. Non-Provisionalapplication Ser. No. 15/640,305, entitled “Systems and Methods forImplementing Time Delay Integration Imaging Techniques in Conjunctionwith Distinct Imaging Regions on a Monolithic Charge-Coupled DeviceImage Sensor”, filed Jun. 30, 2017, which claims priority under 35U.S.C. 119(e) to U.S. Provisional Patent Application Ser. No.62/402,851, entitled “Systems and Methods for Implementing Time DelayIntegration Imaging Techniques in Conjunction with Distinct ImagingRegions on a Monolithic Charge-Coupled Device Image Sensor”, filed Sep.30, 2016 and U.S. Provisional Patent Application Ser. No. 62/405,120,entitled “Systems and Methods for Implementing Time Delay IntegrationImaging Techniques in Conjunction with Distinct Imaging Regions on aMonolithic Charge-Coupled Device Image Sensor”, filed Oct. 6, 2016. Thedisclosures of U.S. Non-Provisional application Ser. No. 15/640,305,U.S. Provisional Patent Application Ser. No. 62/402,851 and U.S.Provisional Patent Application Ser. No. 62/405,120 are herebyincorporated herein by reference in their entirety.

FIELD OF THE INVENTION

The present invention generally relates to the implementation of timedelay integration imaging techniques together with custom filter arraysin the context of charge-coupled device image sensors used to trackmoving objects at high speed.

BACKGROUND

A variety of techniques can be utilized to image scenes in ways thatcapture information within different portions of the visible and/orelectromagnetic spectrum. ‘Color filters’ are often used with a camerasystem to filter out portions of the electromagnetic spectrum with theexception of a specific band, such that only the particularly exemptspectral band is able to transmit through the filter. Thus, for example,a red color filter typically operates to filter out all portions of theelectromagnetic spectrum except for the band corresponding with visiblered light. Color filters are often implemented as patterns of filtersapplied to individual pixels on an image sensor. A common example of afilter pattern is the Bayer filter pattern. A Bayer filter patterntypically includes an array of red, green, and blue color filtersintended to be disposed over a grid of photosensors (e.g. pixels orphotosites), where each color filter is associated with a singlephotosensor. In a Bayer filter, there are usually twice as many greencolor filters as there are red or blue color filters, which is meant tomimic the physiology of the human eye. In a Bayer filter configuration,each respective photosensor is intended to obtain imaging informationconcerning a particular band of the electromagnetic spectrum. Theaggregate of the imaging information can thereafter be ‘demosaiced’ orinterpolated to produce a color image. Note that the term ‘colorfilters’ can also be applicable with respect to those portions of theelectromagnetic spectrum adjacent to the visible light portion, e.g. theultraviolet and infrared portions of the electromagnetic spectrum.

The quality of an image captured by an imaging system is typicallydependent upon the number of photons incident on the pixels of an imagesensor. A variety of techniques can be utilized to increase theintensity of light incident on an image sensor including increasing thesize of the optics to capture more light and/or increasing theintegration or exposure time of the pixels. The extent to which exposuretime can be increased is often limited based upon relative motionbetween the imaging system and the scene and/or motion within the sceneitself. As exposure time increases, scene motion can introduce motionblur artifacts into the resulting image.

Time delay integration (TDI) is an imaging technique that is typicallyimplemented in conjunction with charge-coupled device (CCD) imagesensors for imaging systems that move in a predictable way relative to ascene. CCD image sensors typically operate as follows: (1) a CCD imagesensor typically includes a grid of pixels; (2) when an image of a sceneis desired, electrical charge is stored in the grid of pixels as afunction of the scene's light intensity; (3) the stored electricalcharge is shifted—from one row of pixels to the next—until it reaches aserial register, where stored electrical charge corresponding with eachpixel then proceeds to be read out and stored as image data. Note thatin a conventional CCD imaging technique, each pixel in the grid ofpixels stores light intensity information corresponding with a differentaspect of the scene.

A TDI mode of operation can be useful when it is known that the scene tobe imaged is moving in a known and predictable manner relative to theCCD image sensor. Whereas capturing an image in this scenario using aconventional CCD imaging technique can result in motion blur, capturingan image in this scenario using a TDI mode of operation can increase theexposure time of each resulting image pixel while mitigating thedevelopment of motion blur. In a typical TDI mode of operation, a scenebeing imaged is moving in an “along track” direction—corresponding witha column of pixels. Relatedly, the orthogonal direction is known as the“across track” direction—and it corresponds with a row of pixels. FIG. 1diagrams this configuration. In effect each row of pixels ‘scans’ ascene such that light from a point in the scene impinges on eachsuccessive pixel of a given corresponding column. As the light impingeson each successive pixel, corresponding electrical charge is storedwithin the respective pixel, and the stored electrical charge iscumulatively added to each successive pixel, such that the summation isindicative of all the light that impinged on the column. As can beappreciated, because light from a single point within a scene impingeson the image sensor for a greater duration using a TDI mode of operationrelative to a conventional CCD imaging technique, a TDI mode ofoperation can produce an image characterized by a relatively highersignal to noise ratio. In other words, TDI imaging techniques canprovide for a longer exposure time since light is being aggregated dueto a point in a scene being successively focused onto multiple pixels ina column of pixels due to relative motion of the camera system and thescene, allowing for more photons to be detected, and this can result ina significantly higher signal to noise ratio.

SUMMARY OF THE INVENTION

Systems and methods in accordance with various embodiments of theinvention implement TDI imaging techniques in conjunction withmonolithic CCD image sensors having multiple distinct imaging regions,where TDI imaging techniques can be separately implemented with respectto each distinct imaging region. In many embodiments, the distinctimaging regions are defined by color filters or color filter patterns(e.g. a Bayer filter pattern); data from the distinct imaging regionscan be read out concurrently (or else sequentially and/or nearlyconcurrently). In order to facilitate TDI using different filterpatterns, CCD image sensors in accordance with many embodiments of theinvention enable shifts of multiple rows to support TDI imaging with 2pixel×2 pixel (or larger) filter pattern mosaics. A camera system inaccordance with one embodiment of the invention includes: an opticalsystem; a CCD image sensor, itself including a plurality of pixels thatdefine at least two distinct imaging regions; where the pixels withineach distinct imaging region are configured to operate in unison toimage a scene differently than at least one other distinct imagingregion; a CCD image signal processor; and a microprocessor; where theoptical system is configured to focus incident electromagnetic wavesonto the CCD image sensor; and where the camera system is operable in atime-delay integration mode whereby time delay-integration imagingtechniques are imposed with respect to each distinct imaging region.

A camera system in accordance with an additional embodiment of theinvention includes: an optical system; a CCD image sensor, itselfcomprising a plurality of pixels that define at least two distinctimaging regions, where the pixels within each distinct imaging regionare configured to image a scene differently than at least one otherdistinct imaging region. The camera system also includes: a CCD imagesignal processor; and a microprocessor. In addition, the optical systemis configured to focus incident electromagnetic waves onto the CCD imagesensor; and the camera system is operable in a time-delay integrationmode whereby time delay-integration imaging techniques are imposed withrespect to each distinct imaging region.

In a further embodiment, in a time-delay integration mode, the camerasystem operates to read out accumulated intensity information from eachdistinct imaging region by repeatedly:

accumulating intensity information in pixels within each row of pixelswithin a distinct imaging region;

shifting accumulated intensity information to a next row of pixelswithin the distinct imaging region, where a last row of pixels does nothave a next row of pixels and instead shifts accumulated intensityinformation into an array of accumulators; and

reading out accumulated intensity information from the array ofaccumulators.

Another embodiment also includes: an amplifier operable to amplifyaccumulated intensity information read out from the CCD image sensor;and an analog to digital converter operable to receive amplifiedaccumulated intensity information, and output a digital representationof the accumulated intensity information.

In a still further embodiment, the at least two distinct imaging regionsdefine: a first distinct imaging region configured to image a scene in asingle channel; and a second distinct imaging region configured to imagea scene in multiple channels.

In still another embodiment, the second distinct imaging region employsa filter pattern including color filters selected from the groupconsisting of at least one red color filter, at least one green colorfilter, at least one blue color filter, and at least one yellow colorfilter.

In a yet further embodiment, the second distinct imaging region includesa repeated pattern of filters.

In yet another embodiment, the repeated pattern of filters repeats apattern of filters applied to pixels in a region of at least 2 pixels×atleast 2 pixels.

In a further embodiment again, the repeated pattern of filters repeats apattern of filters applied to pixels in a region of 2 pixels×2 pixels.

In another embodiment again, the pattern of filters applied to pixels ina region of 2 pixels×2 pixels applies the same filter to at least two ofthe pixels and a different filter to at least one of the pixels.

In a further additional embodiment the pattern of filters is a Bayerpattern.

In another additional embodiment, the pattern of filters comprisesfilters that select different wavelengths of red light.

In a still yet further embodiment, the pattern of filters comprisesfilters that select different wavelengths of blue light.

In still yet another embodiment, the pattern of filters comprisesfilters that select different wavelengths of green light.

In a still further embodiment again, the repeated pattern of filters isa repeated pattern of transmissivity filters.

In still another embodiment again, the transmissivity filters aredichroic filters.

A still further additional embodiment also includes a primary filterlocated between the optical system and the CCD image sensor.

In still another additional embodiment, the primary filter is a butcherblock filter layer comprising a plurality of regions possessingdifferent filtering characteristics.

In a yet further embodiment again, the repeated pattern oftransmissivity filters includes filters having different transmissivityto enable high dynamic range imaging.

In yet another embodiment again, the first distinct imaging region isconfigured to image with a fill factor that is smaller than the fillfactor of the pixels in the second distinct imaging region.

In a further additional embodiment again, at least one of the distinctimaging regions defines a third distinct imaging region configured toimage near-infrared electromagnetic radiation.

In another additional embodiment again, at least one of the distinctimaging regions defines a third distinct imaging region configured toimage ultraviolet electromagnetic radiation.

In another further embodiment, shifting accumulated intensityinformation to a next row of pixels within at least one of the distinctimaging regions comprises shifting the accumulated intensity informationto an adjacent row of pixels.

In still another further embodiment, shifting accumulated intensityinformation to a next row of pixels within at least one of the distinctimaging regions comprises shifting the accumulated intensity informationto a non-adjacent row of pixels.

In yet another further embodiment, the camera system is operable toadjust imaging parameters based upon at least one factor selected fromthe group consisting of received positioning information, trajectory,and field of view.

A camera system in accordance with another further embodiment againincludes: an optical system; a CCD image sensor, itself comprising aplurality of pixels that define at least two distinct imaging regions,where the pixels within each distinct imaging region are configured toimage a scene differently than at least one other distinct imagingregion; a CCD image signal processor; a microprocessor; an amplifieroperable to amplify accumulated intensity information read out from theCCD image sensor; and an analog to digital converter operable to receiveamplified accumulated intensity information, and output a digitalrepresentation of the accumulated intensity information. In addition,the optical system is configured to focus incident electromagnetic wavesonto the CCD image sensor; and the camera system is operable in atime-delay integration mode whereby the camera system operates to readout accumulated intensity information from each distinct imaging regionby repeatedly:

accumulating intensity information in pixels within each row of pixelswithin a distinct imaging region;

shifting accumulated intensity information to a next row of pixelswithin the distinct imaging region, where a last row of pixels does nothave a next row of pixels and instead shifts accumulated intensityinformation into an array of accumulators; and

reading out accumulated intensity information from the array ofaccumulators.

In another further additional embodiment, the at least two distinctimaging include an imaging region having a repeated pattern of filtersthat repeats a pattern of filters applied to pixels in a region of atleast 2 pixels×at least 2 pixels; and shifting accumulated intensityinformation to a next row of pixels within the imaging region having therepeated pattern of filters comprises shifting the accumulated intensityinformation to a non-adjacent row of pixels.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the operation of a conventional TDI CCD image sensor.

FIG. 2 illustrates a Satellite Constellation System that can benefitfrom the implementation of TDI imaging techniques in conjunction withmonolithic CCD image sensors characterized by multiple distinct imagingregions in accordance with certain embodiments of the invention.

FIG. 3 conceptually illustrates a CubeSat that can be configured toinclude a camera system that is operable to implement TDI imagingtechniques in conjunction with multiple distinct imaging regions on amonolithic CCD image sensor in accordance with certain embodiments ofthe invention.

FIGS. 4A-4B illustrate a camera system that is configured to implementTDI imaging techniques in conjunction with multiple distinct imagingregions on a monolithic CCD image sensor in accordance with certainembodiments of the invention.

FIG. 5 illustrates a monolithic CCD image sensor that includes amonochrome imaging region and a RGB-NIR, or red-green-blue-nearinfrared, color imaging region, each being compatible with theimplementation of TDI imaging techniques in accordance with certainembodiments of the invention.

FIG. 6A illustrates a monolithic CCD image sensor that includes amonochrome region, a red imaging region, a green imaging region, a blueimaging region, and a yellow imaging region, each being compatible withthe implementation of TDI imaging techniques in accordance with certainembodiments of the invention.

FIG. 6B illustrates a monolithic CCD image sensor that includes multiplenarrowband imaging regions patterned with a mosaic of color filtersformed using a 2 pixel×2 pixel color filter pattern, where eachnarrowband imaging region is compatible with the implementation of TDIimaging techniques in accordance with certain embodiments of theinvention.

FIGS. 6C-6E illustrate 2 pixel×2 pixel color filter patterns that can bemosaicked to form narrowband color filter patterns in accordance withcertain embodiments of the invention.

FIG. 6F illustrates an image sensor incorporating a butcher block colorfilter layer positioned above a monolithic CCD image sensor and dichroicmirror filters formed on a substrate positioned above pixels of the CCDimage sensor in accordance with certain embodiments of the invention.

FIG. 7 illustrates a monolithic CCD image sensor that includes amonochrome region, an RG imaging region, and a B-NIR imaging region,each being compatible with the implementation of TDI imaging techniquesin accordance with certain embodiments of the invention. In particular,columns within each distinct region are associated with a single colorfilter. In this way, the electric charge can be more straightforwardlyaccumulated from each pixel within each column as multiple row shiftsare not required to perform TDI imaging in the illustratedconfiguration.

FIG. 8 illustrates a monolithic CCD image sensor that includes amonochrome region, and an RGB-NIR region, where each of the monochromeregion and RGB-NIR region are characterized by a different number ofrows, each region being compatible with the implementation of TDIimaging techniques in accordance with certain embodiments of theinvention.

FIG. 9A illustrates a monolithic CCD image sensor that includes anHDR+supersampling region, an RGB-NIR color imaging region, an infraredimaging region, and a region for imaging blues, each being compatiblewith the implementation of TDI imaging techniques in accordance withcertain embodiments of the invention.

FIG. 9B illustrates a monolithic CCD image sensor that includes anHDR+supersampling region, a visual bands (RGBM) region, a NIR region,and a region for imaging blues, each being compatible with theimplementation of TDI imaging techniques in accordance with certainembodiments of the invention.

FIG. 9C illustrates a monolithic CCD image sensor that includes a redregion, a green region, a first panchromatic region, a secondpanchromatic region, a NIR region, and a blue region, each beingcompatible with the implementation of TDI imaging techniques inaccordance with certain embodiments of the invention.

FIG. 9D conceptually illustrates monolithic CCD image sensors thatinclude a panchromatic region that can be configured for HDR andsuper-sampling, and a plurality of narrowband regions that are eachpatterned with a mosaic of a dichroic mirror color filter pattern, eachbeing compatible with the implementation of TDI imaging techniques inaccordance with certain embodiments of the invention.

FIG. 9E is a table that conceptually illustrates various spectral bandsof interest that can be selected using color filters associated with animaging region of a monolithic CCD image sensor having multiple distinctimaging regions, each being compatible with the implementation of TDIimaging techniques in accordance with certain embodiments of theinvention.

FIG. 10 illustrates a HDR (high dynamic range) transmissivity filterpattern that can be implemented on a monolithic CCD sensor characterizedby multiple distinct imaging regions, each region being compatible withthe implementation of TDI imaging techniques in accordance with certainembodiments of the invention.

FIG. 11 illustrates the concept of supersampling that can be implementedon a monolithic CCD sensor characterized by multiple distinct imagingregions, each region being compatible with the implementation of TDIimaging techniques in accordance with certain embodiments of theinvention.

FIG. 12 illustrates how camera systems can operate to accumulateelectrical charge in accordance with a TDI imaging technique in eitherof two directions in accordance with many embodiments of the invention.

DETAILED DESCRIPTION

Turning now to the drawings, systems and methods for implementing timedelay integration (TDI) imaging techniques in conjunction with each ofmultiple distinct imaging regions on a monolithic charge-coupled device(CCD) image sensor are illustrated. Modern digital cameras typicallyimplement one of two broadly categorized imaging technologies:Complementary Metal-Oxide-Semiconductor (CMOS) technology and CCDtechnology. The particularly implemented imaging technology for arespective digital camera is largely a function of the intendedapplication for the camera. For example, CCD image sensors havetypically been implemented in satellite imaging applications as they aregenerally more resilient against deleterious radiation effects relativeto conventional CMOS image sensors.

Recently, there has been much interest in developing and manufacturingextremely cost-effective satellites. For example, whereas launching aconventional satellite can cost on the order of $855 million, launchinga ‘CubeSat’ can cost on the order of $150,000. CubeSats' generally referto a type of miniaturized satellite, often used for imaging and/orresearch. They generally have volumes of approximately tens of thousandsof cubic centimeters, and they are often constructed using commercialoff the shelf components.

As can be appreciated, digital cameras based on CCD image sensors can bewell suited for implementation in a CubeSat. However, because of sizeand economic constraints, CubeSats can particularly benefit from theimplementation of more efficient imaging techniques. Accordingly, inmany embodiments of the invention, TDI imaging techniques areimplemented in association with CubeSat camera systems. For example, TDIimaging techniques can be effectively implemented while a respectiveCubeSat orbits in a known manner—e.g. such that a targeted scene movesin a known, predictable manner—to result in the capture of image datacharacterized by relatively higher signal to noise ratio. While much ofthe discussion that follows references CubeSats, one of ordinary skillin the art can readily apprehend that the monolithic CCD image sensorhaving multiple distinct imaging regions, each being compatible with theimplementation of TDI imaging techniques, described herein can beutilized in a variety of imaging applications including (but not limitedto) larger satellite applications and applications in which the sensoris mounted to a moving aerial vehicle and terrestrial applications thatimage scenes moving in a predictable manner.

Notably, TDI imaging techniques have conventionally been implemented inconjunction with a single monolithic CCD image sensor, characterized bya single imaging region—e.g. a grid of pixels in association with aBayer filter arrangement. However, a broader image data set can beefficiently captured with the implementation of a more nuanced CCD imagesensor. Thus, in many embodiments of the invention, camera systemsimplement a monolithic CCD image sensor characterized by multipledistinct imaging regions, each of which is particularly configured toseparately image a scene using different imaging characteristics,whereby TDI imaging techniques can be implemented with respect to eachdistinct imaging region. In this way, a diverse set of image data can beefficiently captured. For example, a monolithic CCD image sensor caninclude: a black and white imaging region; a color imaging region; aninfrared imaging region; and an ultraviolet imaging region. In manyembodiments, narrowband imaging can be performed using a mosaickedpattern of color filters that select for specific wavelengths within anarrow spectral band. When an imaged object has uniform reflectancespectrum, then the pixels in the narrowband imaging region willeffectively form a monochromatic imager. When an imaged object has areflectance spectrum that varies with wavelength, then the pixels in thenarrowband imaging region can be used to capture image data with respectto specific wavelengths at a lower resolution. In several embodiments,acquired image data can be analyzed post-capture and a determinationmade automatically concerning the manner in which to represent theacquired data. As noted above, TDI imaging techniques can be implementedin conjunction with each such region. In many instances, TDI imaginginvolves simply shifting charge between adjacent pixels in the columnsof a CCD image sensor. In a number of embodiments, TDI imaging involvesshifting charge to pixels located multiple rows distant in the columnsof the CCD image sensor. As can readily be appreciated, the specific TDIimaging configuration largely depends upon the color filters utilizedwith respect to a specific image sensor region and the requirements of agiven application. Furthermore, shifts of different numbers of rows canbe applied in different regions of the image sensor. The ability toperform TDI imaging in multiple regions of an image sensor can enableprofound advantages relative to prior imaging techniques. For example,image data acquired in relation to certain of the distinct imagingregions can be correlated with scientific information e.g. the growthrates of imaged vegetation, while image data acquired in relation toanother subset of the distinct imaging regions can be used to moreconventionally image the scene. In this way, a scene can be imaged, andscientific data can be acquired about the scene in a ‘single pass.’

In many embodiments, camera systems are operable to implement TDIimaging techniques in each of two directions (e.g. either ‘up’ columnsof pixels or ‘down’ columns of pixels) for each of multiple distinctimaging regions. In this way, an associated satellite can reverse itsorientation in relation to the sun (e.g. so as to orient its solarpanels to enhance solar flux), and still maintain viable TDIimplementation for each of the multiple distinct imaging regions.

In many embodiments, implemented TDI imaging techniques are controlledby a CCD image signal processor (which in turn is controlled by anonboard microprocessor and/or an FPGA), and the CCD image signalprocessor can dynamically reconfigure TDI imaging techniqueimplementation based on context. For example, as a satelliteelliptically orbits the earth, the relative motion of the earth as it isbeing imaged varies proportionately; accordingly, in many embodiments,the CCD image signal processor can adjust the parameters forimplementing TDI imaging techniques to accommodate this variation. Forinstance, the rate at which accumulated charge is shifted between therows of the CCD can be increased (i.e. overall integration time can bereduced) when the earth is moving relatively fast in the field of viewof the respective camera system, and the rate at which accumulatedcharge is shifted between the rows of the CCD can be decreased (i.e.overall integration time may be increased) when the earth is movingrelatively slow in the field of view of the respective camera system.The change in rate need not necessarily correspond to a change inintegration time. In many embodiments, integration time can becontrolled relative to the rate at which accumulated charge is shifted.In addition, additional rows can be provided to adjust the number ofrows that contribute to the accumulated charge based upon the rate atwhich accumulated charge is shifted between the rows of the CCD.

For context, imaging satellite systems that can benefit from thesecamera systems are now discussed below in greater detail.

Imaging Satellite Systems

In many embodiments, imaging satellite systems are implemented thatinclude a constellation of satellites, at least one of which includes acamera system that utilizes a monolithic CCD image sensor characterizedby multiple different imaging regions, and operable to implement TDIimaging techniques with respect to each of the multiple differentimaging regions. In many instances, such satellite constellations can beused to aggregate image data that can eventually be accessed by clientdevices. The satellites can interface with terrestrial systems to relayimage data in any of a variety of configurations. Thus, for instance,FIG. 2 illustrates an imaging satellite system—whereby at least onesatellite includes a camera system that utilizes a monolithic CCD imagesensor characterized by multiple different imaging regions, and operableto separately implement TDI imaging techniques with respect to eachdistinct imaging region—that interfaces with a single ground station. Inparticular, FIG. 2 illustrates a constellation of satellites operable tocollect image data. At least one satellite includes a camera system thatutilizes a monolithic CCD image sensor characterized by multipledistinct imaging regions and is operable to implement TDI imagingtechniques with respect to each distinct imaging region. The particularsof implementing TDI imaging techniques will be discussed in greaterdetail in subsequent sections. Any of a variety of satellite classes canbe implemented in accordance with various embodiments of the invention.For example, in many embodiments, CubeSats are implemented. In a numberof embodiments, more conventional satellites are implemented. As notedabove, imaging systems in accordance with many embodiments of theinvention can be mounted to any of a variety of classes of vehicleincluding (but not limited to) aerial vehicles and/or utilized withinterrestrial applications.

FIG. 2 illustrates that the constellation of satellites interacts with aGround Station. Thus, for instance, the constellation of satellites canrelay acquired imaging data as well as respective positioninginformation to the Ground Station. The Ground Station can be used tocommunicate with the constellation of satellites generally, and morespecifically to control the trajectory and operation of the varioussatellites within the constellation. Thus, for example, FIG. 2illustrates that a Mission Control center can be used to interact withthe Ground Station and thereby control the operation of theconstellation of satellites. Mission Control can be in wirelesscommunication with the Ground Station or in wired communication with theGround Station.

The Ground Station can also serve to relay received image data to animage data database. As before, the Ground Station can be in wirelesscommunication with the servers that ingest data into the image datadatabase or wired communication with the image data database. The imagedata database can then store the image data for subsequent use. Forinstance, the image data can be retrieved and processed by a serversystem that provides access to the image data to any of a variety ofclient applications, e.g. via the Internet. While the accessing of imagedata over the Internet is depicted, it should be clear that image datacan be accessed via any suitable network. For example, in someinstances, it can be accessed over a local area network. As can beappreciated, all data communications can be encrypted for security.

The above description has provided one example of an imaging satellitesystem that can be implemented that utilizes monolithic CCD camerasensors, characterized by multiple different imaging regions, andoperable to implement TDI imaging techniques with respect to eachdistinct imaging region. But it should be appreciated that such systemscan be implemented in any of a variety of configurations. For example,in many embodiments, multiple ground stations can be utilized tointerface with the constellation of satellites. For instance, themultiple ground stations can be located around Earth so that satelliteswithin the constellation can always have a line of sight to at least oneground station. In many embodiments, the satellites within theconstellation are operable to form a mesh network, whereby thesatellites can communicate with each other. Thus, for example,satellites can relay imaging data to one another, and also to a GroundStation. This configuration can allow a satellite to relay image data toa Ground Station even if the Ground Station is not within a line ofsight of a satellite. By way of example, the satellite can relay imagedata to a second satellite that is within line of sight of the GroundStation, and the second satellite can thereafter relay the image data tothe target Ground Station. Similarly, a Ground Station can communicatewith a satellite that it does not have direct line of sight to using themesh network. In this way, a mesh network can allow for operation usingrelatively fewer ground stations (e.g. since the satellites can functionas communication relays).

While a particular configuration has been illustrated, and variants havebeen discussed, it should be clear that any suitable system forimplementing a constellation of satellites that implements at least onecamera system that utilizes a monolithic CCD image sensor characterizedby multiple distinct imaging regions, whereby TDI imaging techniques canbe implemented with respect to each region, can be implemented inaccordance with many embodiments of the invention. Individual satellitesthat can include a camera system that utilizes a monolithic CCD imagesensor characterized by multiple different imaging regions, and that isoperable to implement TDI imaging techniques on each of the multipleregions is discussed in greater detail below.

Satellites Including Camera Systems Utilizing Monolithic Ccd ImageSensors Having Multiple Tdi Technique-Ready Imaging Regions

In many embodiments, satellites are implemented that include camerasystems that utilize monolithic CCD image sensors having distinctimaging regions, where TDI imaging techniques can be implemented withrespect to each distinct imaging region. Notably, the satellites can beimplemented in any of a variety of form factors. For example, in manyembodiments CubeSats are implemented that include the described camerasystems. In a number of embodiments, the camera systems are implementedwithin more conventional satellites.

FIG. 3 illustrates an example of a CubeSat that can be configured toinclude a camera system having a monolithic CCD image sensorcharacterized by multiple distinct imaging regions, where TDI imagingtechniques can be implemented with respect to each distinct imagingregion. In particular, it is depicted that the CubeSat 300 includes thecamera system 302, a housing for the camera system 304, an antenna 306,and solar panel extensions 308. As can be appreciated, the camera system302 can generally include the optics, a monolithic CCD image sensorcharacterized by multiple distinct imaging regions, and associatedcontrollers and processors that can enable the implementation of TDIimaging techniques for each of the multiple distinct imaging regions.The particulars concerning the operation of the camera system aredescribed in subsequent sections.

The housing 304 for the optical system can be made of any suitablematerial in accordance with various embodiments of the invention. Inmany embodiments, the housing for the optical system comprises materialthat is radiation resistant. In many embodiments, the housing furtherincludes adjoined solar panels that can be used to provide additionalpower for the CubeSat.

As can be appreciated, the antenna 306 can allow for communication, e.g.with other satellites and/or terrestrial-based stations. Thecommunication can be performed in accordance with any suitable protocol,including e.g. via RF communication.

As can be appreciated, the solar panel extensions 308 can comprise anysuitable material in accordance with a number of embodiments of theinvention. In many embodiments, the solar panel extensions 308 arecompactable. Thus, as the CubeSat is being launched into outer space,the CubeSat can adopt a compacted configuration whereby the solar panelextensions 308 are folded tight against the housing for the camerasystem 304; and they can subsequently deploy when the CubeSat isreleased into orbit. As can be appreciated, this operability can enablevolumetric space saving, which can subsequently allow a respectivelaunch vehicle to carry more payload (e.g. a constellation of suchCubeSats) for efficient delivery into orbit.

The alluded to camera systems including monolithic CCD image sensorscharacterized by multiple distinct imaging regions, which are configuredto implement TDI imaging techniques with respect to each distinctimaging region, in accordance with many embodiments of the invention arenow discussed in greater detail below.

Camera Systems Including Monolithic CCD Image Sensors Having MultipleTDI-Ready Imaging Regions

In many embodiments, camera systems are implemented that utilizemonolithic CCD imagers including multiple distinct imaging regions andare operable to implement TDI imaging techniques with respect to eachdistinct imaging region. Such camera systems can be implemented in awide variety of contexts, including e.g. satellites, including CubeSats,as discussed above, and/or additional contexts including (but notlimited to) telescopes, and mounting to drones, or airplanes.Accordingly, camera systems in accordance with various embodiments ofthe invention are not limited to any specific application.

FIGS. 4A and 4B illustrate the architecture for a camera system that canbe implemented in accordance with many embodiments of the invention. Inparticular, FIG. 4A depicts that the camera system 400 includes optics402, a primary filter 404, a secondary filter 406, and the aggregate ofthe CCD image sensor and associated circuitry 408. As can beappreciated, the optics 402 can include optical element(s) that functionto focus electromagnetic waves on the CCD image sensor. The primaryfilter 404 can function to filter out unwanted electromagneticwavelengths. The illustrated embodiment further includes a secondaryfilter 406; the secondary filter 406 can function to implement e.g. aBayer filter or any other color filter pattern as desired. The manner inwhich a primary filter including different regions that select differentspectral bands that are then further filtered by specifically tunedfilters in a secondary filter is discussed further below with referenceto FIG. 6F. While several examples of the types of filters that can beimplemented are mentioned, any suitable filters can be implemented inaccordance with various embodiments of the invention. In someembodiments, only a primary filter is implemented. In many embodiments,no separate discrete filter is implemented. FIG. 4A further depicts theaggregate of the monolithic CCD image sensor and associated electronics408, which are responsible for perceiving impinging electromagneticwaves.

FIG. 4B depicts in greater detail one embodiment of the aggregate of theCCD image sensor and associated electronics 408. In particular, FIG. 4Bdepicts that electromagnetic waves (having been filtered by the optics)impinge onto the constituent CCD image sensor 410. The CCD image sensor410 can be controlled by a FCCD image signal processor 412. As can beappreciated, the CCD image sensor 410 can be controlled to implement TDIimaging techniques in accordance with various embodiments of theinvention. In many embodiments, in implementing TDI imaging techniques,the CCD image signal processor can use positional information incomputing the integration time and other imaging parameters to beimplemented. For example, if the CubeSat is circumnavigating the earthin an elliptical orbit while imaging the earth, the integration time tobe implemented will be different based on the position of the CubeSatrelative to the elliptical orbit.

In a number of embodiments, a filter pattern is applied to an imagingregion on the CCD image sensor 410 that is constructed using a mosaic ofa smaller filter pattern (e.g. a 2 pixel×2 pixel pattern, a 2 pixel×3pixel pattern, a 3 pixel×2 pixel pattern, a 3 pixel×3 pixel pattern, a 4pixel×4 pixel pattern, and or any generalized m pixel×n pixel pattern).Where different color filters are applied to adjacent pixels in acolumn, the FCCD image signal processor 412 can control the shifting ofaccumulated charge to shift the accumulated charge by multiple rows ofpixels. In this way, TDI is performed so that accumulated charge ismoved within a column between pixels that share a common type of filter.In a number of embodiments, different regions incorporate differentfilter patterns and can be controlled so that accumulated charge isshifted by a specific number of rows dependent upon the imaging region.The implementation of TDI imaging techniques with respect to each ofmultiple distinct imaging regions are discussed in greater detail below.

In many embodiments, an amplifier 414 is implemented to amplify thesignal output by the CCD image sensor. The amplified signal can then besent to an analog to digital converter 416 that can convert the signalto a digital form. The digital data can then be stored in data storage420. In many embodiments, the data can be encoded using a lossy orlossless encoding prior to storage of the encoded data. In particular,it is illustrated that a microprocessor 418 can be used to control theoperation of the data storage as well as the operation of the CCD imagesignal processor. The image data can then be relayed to a ground stationvia an associated antenna (not shown) for subsequent consumption.

While a specific camera system architecture has been illustrated anddiscussed, camera systems having CCD image sensors characterized bymultiple distinct imaging regions and operable to implement TDI imagingtechniques with respect to each of the distinct regions can beimplemented in any of a variety of ways in accordance with variousembodiments of the invention. The implementation of distinct imagingregions within monolithic CCD image sensors, and the implementation ofTDI imaging techniques with respect to each imaging region in accordancewith many embodiments of the invention is discussed below.

TDI Imaging Techniques with Respect to Each of Multiple Distinct ImagingRegions in a Monolithic CCD Image Sensor

In many embodiments, monolithic CCD image sensors that include multipledistinct imaging regions, whereby TDI imaging techniques can be utilziedwith respect to each distinct imaging region, are implemented. Suchsensors can be implemented to efficiently acquire a diverse and robustset of image data. Notably, any of a variety of distinct imaging regionscan be implemented in accordance with various embodiments of theinvention. Within the context of CCD image sensors, a distinct imagingregion can be understood to be a grid of pixels that are configured toseparately image a scene according to a set of imaging parameters, andwhich can support the viable and sensible application of TDI imagingtechniques. As an example, imaging regions can be defined by filtersand/or a filter pattern. For instance, a grid of pixels can beassociated with infrared color filter(s) and an adjacent grid of pixelscan have different imaging characteristics including use of a differentcolor filter and/or filter pattern. The differences in imagingcharacteristics including (but not limited to) the color filters and/orfilter patterns can define a distinct imaging region. In many instances,an imaging region is defined by a grid of pixels to which a Bayer filterpattern is applied. Thus, a CCD image sensor may include a first imagingregion characterized by a Bayer filter pattern, and a second imagingregion characterized by an ability to image the infrared portion of theelectromagnetic spectrum. Importantly, data from the distinct imagingregions can be read out concurrently (or else sequentially and/or nearlyconcurrently). In many embodiments, the CCD image sensor includesmultiple ports and an external microcontroller and or FPGA is able toread out the accumulated intensity information from a row of pixels ineach of the distinct regions during the time each row of pixels isexposed prior to shifting accumulated intensity information to a nextrow of pixels.

Moreover, the different imaging regions can be characterized by varying‘heights,’ i.e. different imaging regions can be characterized bydifferent numbers of rows of pixels. More in detail, a distinct imagingregion that includes more rows can be associated with a longerintegration time, i.e. electric charge is accumulated over more rows ofpixels. Thus, for instance, if one of the distinct imaging regions isgenerally not as sensitive to incident electromagnetic radiation and/orless photons within a portion of the electromagnetic spectrum areincident on the image sensor, the region that images that portion of theelectromagnetic spectrum can have more rows to allow for a longerintegration time.

When a filter pattern is used within an imaging region in whichdifferent filters are applied to adjacent pixels within a column, theCCD can be configured to shift accumulated charge by multiple rows at atime so as to skip over pixels having different filters. As differentregions can have different filter patterns, the CCD can be controlled toapply different shifts in different regions. Furthermore, theintegration time can be determined as both a function of the number ofrows within a region and the number of rows by which accumulated chargeis shifted.

FIG. 5 illustrates a monolithic CCD image sensor divided equally intotwo distinct imaging regions: a monochrome region and an RGB-NIR region,in accordance with an embodiment of the invention. The CCD image sensorcan be configured such that each distinct region can image a respectivescene separately, and TDI imaging techniques can be implemented withrespect to each distinct imaging region. As alluded to above, byincluding multiple distinct regions, relatively more diverse,information-rich image data can be obtained.

Importantly, while TDI imaging techniques can be performed on themonochrome region relatively straightforwardly, implementing TDI imagingtechniques on the RGB-NIR region depicted in FIG. 5 may require morenuance. In particular, rather than accumulating electrical charge fromeach pixel in a respective column, the electrical charge from pixels inalternating rows, which correspond to the same color filter (e.g.green), may be aggregated to prevent the blurring of different colorinformation.

In some embodiments, to obtain color information without having toimplement these nuanced TDI-imaging techniques, red, green, blue, andnear infrared color filters define their own respective imaging regions.Thus, for instance FIG. 6A illustrates an embodiment that includes fivedistinct imaging regions: a monochrome region, a red region (R), a greenregion (G), a blue region (B), and a near-infrared region (Y). As can beappreciated from the discussion above, TDI imaging techniques can beimplemented with respect to each of the five distinct regions. In thisconfiguration, electrical charge from each pixel within a column of arespective region can be aggregated in accordance with TDI imagingtechnique principles (as opposed to aggregating only those inalternating rows). Separating the different spectral bands intodifferent regions can involve tradeoffs. The separation increasesresolution, however, it can decrease integration time relative to thesame number of rows of pixels. In addition, the separation cannecessitate readout of accumulated intensity information from more rowsof pixels during the time period in which each row of pixels accumulatesintensity information.

In many embodiments, an ability to image within a specific spectral bandand to provide image data with respect to specific wavelengths ofinterest within a spectral band can be achieved by patterning an imagingregion with a pattern of color filters. A CCD image sensor in whichmultiple image regions are defined based upon specific 2 pixel×2 pixelpatterns of filters in accordance with an embodiment of the invention isillustrated in FIG. 6B. In the illustrated embodiment, a near infraredregion is provided in which a single type of filter is applied to eachpixel within the imaging region. A 430-580 nm imaging region is alsoprovided in which a 2 pixel×2 pixel pattern of filters is utilized toimage specific wavelengths within the 430-580 nm spectral band. Anexample of a pattern of color filters that can be applied to a 2 pixel×2pixel array of pixels within a 430-580 nm imaging region in accordancewith an embodiment of the invention is shown in FIG. 6D. The 2 pixel×2pixel array includes two pixels to which a 490 nm filter is applied, asingle pixel having a 560 nm filter, and a single pixel having a 443 nmfilter.

Referring again to FIG. 6B, the CCD image sensor also includes a 650-950nm imaging region in which a 2 pixel×2 pixel pattern of filters isutilized to image specific wavelengths within the 650-950 nm spectralband to image wavelengths at the edges of the red spectral band. Anexample of a pattern of color filters that can be applied to a 2 pixel×2pixel array of pixels within a 650-950 nm imaging region in accordancewith an embodiment of the invention is shown in FIG. 6C. The 2 pixel×2pixel array includes two pixels to which a 665 nm filter is applied, asingle pixel having a 865 nm filter, and a single pixel having a 945 nmfilter. The CCD image sensor also includes a 650-950 nm imaging regionin which a 2 pixel×2 pixel pattern of filters is utilized to imagespecific wavelengths within the 650-950 nm spectral band to image otherwavelengths in the red spectral band. An example of a pattern of colorfilters that can be applied to a 2 pixel×2 pixel array of pixels withina 650-950 nm imaging region in accordance with an embodiment of theinvention is shown in FIG. 6C. The 2 pixel×2 pixel array includes twopixels to which a 705 nm filter is applied, a single pixel having a 783nm filter, and a single pixel having a 730 nm filter.

As noted above, the advantage of using filters within a spectral bandthat select for specific wavelengths of light is that the image sensorcan image objects that have a uniform reflectance spectrum at a highresolution (i.e. at a sampling rate defined by the pixel pitch of thesensor). In addition, the image sensor can also capture image dataconcerning the reflectance of specific informative wavelengths whenimaging objects that have reflectance spectrums that vary meaningfullywith wavelength. The extent to which variation in reflectance yieldsmeaningful information can be ascertained during post processing byperforming 2D Fourier analysis of the frequency spectrum of the captureddata. Objects that have reflectance spectrums that vary with wavelengthwill tend to have frequency spectra with side lobes. When side lobessatisfying a predetermined criterion are present, the system can flag tothe user and/or present the user with a visualization of the capturedimage data that highlights the different wavelengths within a spectralband. When the criterion is not satisfied, a blind deconvolution can beperformed to present the image data at the resolution of the imagesensor. In several embodiments, processing circuitry within the camerasystem performs a transformation such as, but not limited to, a discretecosine transformation during the encoding of the image data that can befurther utilized to evaluate the spectral characteristics of theacquired image data. In other embodiments, the spectral characteristicsof the acquired image data can be determined post capture and on aremote system.

In a number of embodiments, an optical system utilized with a CCD imagesensor in which a region employs a filter pattern similar to thosedescribed above is designed to have a Q, which is a function of thefocal length of the optics and the pixel pitch of the senor, ofapproximately 1. When the narrowband imaging region of the image sensoracts as a monochromatic imaging region, the electro-optical imagingsystem has a Q of approximately 1. When the narrowband imaging region ofthe image sensor acts as an imaging region with pixels that imagemultiple wavelengths within the narrowband based upon the pattern offilters (due to the variation in the reflectance spectrum of an imagedobject), then the electro-optical imaging system has a Q that isdetermined by the size of the filter pattern. For example, a 2×2 filterpattern can have a Q of approximately 0.5. As can readily beappreciated, the characteristics of an electro-optical imaging systemformed by an optical system and a CCD image sensor is largely dependentupon the requirements of a given application. Furthermore, CCD imagesensors are not limited to filters tuned to select the specificwavelengths described above. Any of a variety of filter patternsincluding 2 pixel×2 pixel, 2 pixel×3 pixel, 3 pixel×2 pixel, 3 pixel×3pixel, 4 pixel×4 pixel and/or any other filter arrangements that selectfor any of a number of different wavelengths that are appropriate to therequirements of a given application can be utilized in accordance withvarious embodiments of the invention.

In a number of embodiments, filter patterns are applied to a CCD imagesensor using dichroic films. In many embodiments, a primary butcherblock filter is paired with the secondary dichroic films so that thelight incident on the dichroic films in band limited. In this way, thebutcher block filter can restrict the spectral band of incidentradiation and the specific dichroic filter applied to an individualpixel can selectively admit a narrowband of light to the pixel. A CCDimage sensor incorporating a primary butcher block filter and asecondary filter constructed using dichroic mirrors in accordance withan embodiment of the invention is illustrated in FIG. 6F. The CCD imagesensor assembly 600 includes a butcher block filter layer 602incorporating a number of different color filter regions that filterincident radiation to pass a specific spectral band. In the illustratedembodiment, the butcher block filter layer 602 includes a first filterregion 604 configured to pass a spectral band of 430-580 nm and a secondfilter region 606 configured to pass red light in a spectral band of 650nm to 950 nm. Radiation filtered by the butcher block filter layer 602is incident upon a CCD image sensor relative to which filter patternsare formed using dichroic mirrors creating imaging regions similar tothose illustrated in FIG. 6B. In the illustrated embodiment, thedichroic mirrors form PAN filtering regions 610, a near infrared imagingregion 612, a 430-580 nm imaging region 614, a first 650-950 imagingregion 616, and a second 650-950 nm imaging region 618. The secondfilter region 606 of the butcher block filter layer 602 acts as aprimary filter for both imaging regions 616 and 618 that image in the650 nm-950 nm band. In the illustrated embodiment, the dichroic mirrorcolor filters are formed on a quartz substrate that is aligned with theCCD image sensor 608.

While a specific CCD image sensor assembly, and variations thereof, aredescribed above with reference to FIG. 6F, any of a variety of imagesensor assemblies can be constructed including CCD image sensorsincluding a single or multiple layers of filters and/or any of a varietyof imaging regions and/or filter types or configurations can be utilizedas appropriate to the requirements of specific applications inaccordance with various embodiments of the invention.

FIG. 7 illustrates yet another embodiment whereby three distinct imagingregions are defined: a monochrome region, a RG region, and a B-NIRregion. In this embodiment, each of the pixels in the same column areassociated with a same color filter type; accordingly, electrical chargefrom each pixel within a same column in a respective imaging region canbe accumulated. As can readily be appreciated, combining pixels fromdifferent spectral bands in a single row can decrease the resolution ofresulting images. The decrease in resolution, however, can come at thebenefit of doubling the period of time in which intensity information isaccumulated via TDI imaging techniques.

As can be gleaned from the above examples, the distinct imaging regionsdo not have to be characterized by equal heights. Rather, each distinctimaging region can include any appropriate number of rows. As alluded toabove, a ‘taller’ imaging region can allow for more integration/exposuretime for a respective imaging region and/or imaging of discretewavelengths within a spectral band using a mosaicked filter pattern.Thus, for instance, if a particular distinct imaging region ischaracterized by a low sensitivity, it can be made to include more rowsof pixels and thereby allow a longer integration time that cancounteract the relatively low sensitivity. For example, in manyembodiments, monolithic CCD image sensors include a first distinctimaging region operable to image that portion of the electromagneticspectrum corresponding with near-IR, and a second distinct imagingregion operable to image the visible light portion of theelectromagnetic spectrum; and the first distinct imaging region is madeto have three times as many rows as the second distinct imaging region.This can allow the first distinct imaging region to have an exposuretime that is three times as long as the second distinct imaging region.FIG. 8 illustrates yet another example of distinct imaging regions beingcharacterized by different numbers of rows. In particular, FIG. 8depicts a monolithic CCD image sensor divided into two unequal regions:a narrower monochrome region, and a wider RGB-NIR region.

While FIGS. 5-8 depict a monolithic CCD image sensor characterized bymonochrome and color imaging regions, CCD image sensors can include anytype of imaging regions in accordance with various embodiments of theinvention. For example, FIG. 9A illustrates a monolithic CCD imagesensor characterized by four distinct imaging regions in accordance withcertain embodiments of the invention: an RGB-NIR region; an IR region; aregion for imaging blues; and an HDR+‘supersampling’ region. The RGB-NIRregion can acquire color information, while the infrared region and theblues region can acquire infrared and ultraviolet informationrespectively. The HDR+supersampling region can acquire image data thatcan be used to construct a high dynamic range image, and also acquiresupersampled image data (HDR imaging and supersampling are discussed ingreater detail below). A similar CCD image sensor characterized by fourdistinct regions in accordance with various embodiments of the inventionis shown in FIG. 9B. The specific imaging regions include anHDR+‘supersampling’ region, a RGB region for imaging visible light, anNIR region including a mosaicked 2 pixel×2 pixel pattern of filters anda region for imaging blues including a mosaicked 2 pixel×2 pixel patternof filters.

While FIGS. 9A and 9B illustrate distinct imaging regions for visiblelight, infrared, and blues, in many embodiments, distinct imagingregions for imaging a narrowband of the electromagnetic spectrum can beimplemented, e.g. a sub-portion of the infrared region of theelectromagnetic spectrum. For example, some narrowband portions of theelectromagnetic spectrum may be correlated with scientific data;accordingly distinct imaging regions can be implemented targeting theseportions. A CCD image sensor including distinct regions for imaging red,green, near infrared, and blue wavelengths and two regions forperforming panchromatic imaging with different exposure times to achievehigh dynamic range imaging in accordance with an embodiment of theinvention is illustrated in FIG. 9C. In the illustrated embodiment, theexposure time varies with region based upon the anticipated intensity oflight within imaged scenes and/or the narrowness of the spectral bandsimaged by specific pixels within an imaging region. The architectureillustrated in FIG. 9C can be generalized to any number of imagingregions having different exposure times as shown in FIG. 9D, which alsoincludes an HDR+‘supersampling’ region. As can be appreciated, thevariety in the types of imaging region that can be provided upon a CCDimage sensor in accordance with various embodiments of the invention arelargely only restricted by the number of rows provided on the CCD imagesensor, the number of lanes from which pixel data can be buffered/readout concurrently and/or the requirements of a given application.Examples of specific spectral bands that may be of interest in variousimaging applications and for which imaging regions can be created on aCCD image sensor in accordance with any of a number of differentembodiments of the invention are provided in the table shown in FIG. 9E.

The Red Edge I, Red Edge II, Near Infrared I, Near Infrared II, and NearInfrared III bands listed in FIG. 9E can open up many new use cases inagriculture and natural resources. The Costal Blue or Coastal Aerosolbands can be useful for counting whale populations, bathymetry, oceancolor observation, marine vegetation, monitoring chlorophyllconcentrations and suspending sediments in water, as well asphytoplankton and algae blooms. In addition, these bands can besensitive to clouds, smoke, and haze and can be used for filtering outimages in which a scene is occluded by clouds during image processing.The Water Vapor band listed in FIG. 9E is a band in which light isstrongly absorbed by water vapor and can be useful for performingatmospheric correction and estimating water vapor content.

While specific spectral bands are described above and with reference toFIG. 9E, filters can be utilized to form an imaging region in which TDItechniques can be utilized to acquire useful information from anyappropriate spectral band in accordance with various embodiments of theinvention. Generally, as can be appreciated, the pixels within eachconstituent region can be specifically adapted to receive respectiveelectromagnetic waves. In this way, higher quality image data can beobtained at the individual pixel level.

Referring again to the various CCD image sensors above, many refer toimage regions that incorporate filters that enable HDR imaging. FIG. 10illustrates an HDR filter that can be applied in conjunction with atleast one distinct imaging region in accordance with an embodiment ofthe invention. In particular, FIG. 10 illustrates a mosaic of high,medium, and low transmissivity filters. Such a filter can be used inconnection with any of a variety of distinct imaging regions, includingthose defined by an RGB-NIR color filter pattern, those adapted to imagethe infrared portion of the electromagnetic spectrum, those adapted toimage the ultraviolet portion of the electromagnetic spectrum, and thoseadapted to perform monochrome imaging. The varying transmissivity canresult in the acquisition of a more diverse set of image data, which canbe used to construct a high dynamic range image. In some embodiments, aCCD image sensor includes two RGB-NIR regions, one of which is furtherassociated with an HDR filter. While an HDR filter characterized by low,medium, and high transmissivity filters is depicted, it should be clearthat any suitable HDR filter can be implemented in accordance withembodiments of the invention. For example, in some embodiments, HDRfilters are characterized by filters of two transmissivity values. Insome embodiments, a mosaiced HDR filter is applied uniformly across theentirety of the monolithic CCD image sensor. In some embodiments,HDR-like image data can be captured without the use of a discrete HDRfilter by varying the exposure time for individual pixels. For example,where greater tranmissivity is desired for a particular pixel, thatpixel can image for a greater duration; this will allow moreelectromagnetic radiation to impinge on the pixel. Similarly, wherelesser transmissivity is desired, a respective pixel can be made toimage for a shorter duration. In this way, greater diversity in imagedata can be achieved. As can be appreciated, the monolithic CCD imagesensor can operate to accumulate charge only of pixels similarlydisposed, e.g. having a same color filter type and having a sametransmissivity filter type.

FIG. 11 illustrates the application of supersampling techniques. Inother contexts, the term supersampling is utilized to describe theintegration of accumulated charge by multiple pixels in a CCD imagesensor during TDI. It should be appreciated that the term supersampling,as utilized herein, instead refers to sampling using a fraction of theactive area of a pixel on the CCD image sensor during TDI to achieveincreased resolution. In FIG. 11 pixels 1102 and 1108 are highlighted.When light impinges on pixel 1102, the pixel is manipulated such thatthe pixel responds to light impinging on one half of the pixel 1104; thepixel does not react to light impinging on the other half of the pixel1106. In other words, the pixel is being biased and/or the pixel isfabricated at a reduced fill factor and/or is masked using a lightblocking material such that it only detects incident photons using afraction (e.g. half) the area of the pixel 1104. Subsequently, inaccordance with the TDI Imaging protocol, when the adjacent pixel 1108is illuminated by the same aspect of the scene, the pixel 1108 respondsto light impinging the opposite half of the pixel 1110. Relatedly, anylight impinging on the other half 1112 does not result in theaccumulation of charge. This pattern can be repeated for thecorresponding row of pixels. As can be appreciated, when the storedelectric charge is accumulated, it is done so for correspondingalternating rows. In this way, it can be seen that by controlling and/orfabricating the pixels in this manner, the resolution of the respectivedistinct imaging region can be effectively doubled.

In many instances, to facilitate supersampling, a respective monolithicCCD image sensor is associated with compatible optics. For example, themodulation transfer function (MTF) of the optics can be relativelyincreased so that the MTF is sufficient to resolve detail at a halfpixel width. This may introduce aliasing into the image data captured byany regions that are not performing supersampling. This can be mitigatedby implementing optics that have varying MTF, or through imageprocessing.

The depicted supersampling mechanics can be implemented by any of theconstituent imaging regions within a monolithic CCD image sensor inaccordance with various embodiments of the invention. In someembodiments, the entire CCD image sensor is operable to implementsupersampling. In many embodiments, supersampling is implemented withrespect to only certain of the distinct imaging regions. In manyembodiments, supersampling is performed in conjunction withtransmissivity filters, such as the HDR filter pattern seen in FIG. 10.

As can be appreciated, the above-described systems and techniques arebroad, and enable much flexibility. For example, a monolithic CCD imagesensor can include any number of constituent imaging regions inaccordance with various embodiments of the invention. Moreover, they caneach be characterized by differing (or the same) numbers of rows and/orshifts of different (or the same) numbers of rows during TDI. And theycan further be implemented in conjunction with transmissivity filtersand/or supersampling techniques. Accordingly, it can be appreciated thatthe described systems and techniques can enable robust imaging, and canbe used to acquire information-rich data.

In many embodiments, such camera systems are further operable toimplement TDI imaging techniques in each of two directions (e.g. either‘up’ columns of pixels or ‘down’ columns of pixels) for each of multipledistinct imaging regions, and this aspect is discussed in greater detailbelow.

Configurable TDI Imaging Technique Implementation

In many instances, satellites rely heavily (if not exclusively) on solarpower. Accordingly, in many instances it may be requisite for arespective satellite to invert its orientation in relation to the Sun soas to enhance incident solar flux. In many cases, this reorientation mayinvolve rotating the camera system's field of view relative to theearth. FIG. 12 illustrates this circumstance. In particular, it isillustrated that at a first time t0 as a satellite is orbiting earth,its onboard solar panels are directly face the sun. As it orbits, attime t1, the solar panels are no longer facing the sun; consequently, attime t2, the satellite reorients so that its solar panels face the sun,and thereby enhances power generation. As can be seen, this can involveinverting the field of view of the camera system as it is imaging theearth. And as can be appreciated, unless mitigating steps are taken,this can disrupt TDI imaging technique protocols. Accordingly, in manyembodiments of the invention, camera systems are implemented whereby TDIimaging techniques can be implemented in each of two directions (e.g.‘up’ columns of pixels or ‘down’ columns of pixels). In this way, thecamera system can accommodate such a reorientation of the camera system.In many embodiments, this can be achieved by having the CCD image signalprocessor configure the CCD image sensor to either shift the accumulatedintensity information up the column of pixels (i.e. in a firstdirection) or down the column of pixels (i.e. in a second oppositedirection) as appropriate in accordance with TDI imaging techniques. Thedetermination of whether to shift the charge up or down the column ofpixels can be based on positional information in relation to the sun.

In many embodiments, the described camera systems can be made furtherrobust by incorporating adaptability with respect to imaging parameters,and this aspect is discussed in greater detail below.

Adaptable TDI Imaging Techniques

In many embodiments, camera systems are configured to augment theimplemented TDI imaging techniques based on applicable context. In thisway, imaging techniques can be made to be more efficient, and image datacan be viably acquired irrespective of lighting and/or satellitetrajectory. For example, in many embodiments, a camera system is made tooperate in conjunction with ‘lookup tables,’ which can contain suitableimaging parameters based on satellite location, trajectory, field ofview, and/or any other relevant parameters. The ‘lookup tables’ can bestored onboard within the satellite, or can be accessed via acommunication link with a ground station. For example, where it is knownthat a camera system is imaging a relatively bright location, a ‘lookuptable’ may be used in adjusting the camera system to implementrelatively short integration times, since longer exposures may not benecessary to receive sufficient electromagnetic radiation. In oneembodiment, the lookup tables are generated on-orbit based on rangingand geolocation information derived from sensor data, GPS readingsand/or ground priors. Note that a shorter integration time may besuitably associated with accumulating intensity information over fewerrows for a respective imaging region. In addition and/or alternatively,the same number of rows can be utilized to accumulate intensityinformation via TDI. However, the period of time over which each pixelaccumulates charge is reduced relative to the rate at which accumulatedintensity information is shifted between rows of pixels. On the otherhand, where it is known that a camera system is imaging a relatively dimlocation, the camera system may be adjusted to implement longerintegration times, so as to enable more suitable imaging accounting forthe relatively low light. As can be appreciated, the imaging parameterscan also be based on ‘time of day,’ i.e. presence of sunlight on thescene being imaged: where more sunlight is present, shorter integrationtimes may be used.

In many embodiments, TDI imaging parameters can be adjusted based on theposition and trajectory within an elliptical orbit. Of course, it shouldbe appreciated that while several relevant parameters are mentioned, theimaging parameters of a camera system can be adjusted based on anyrelevant parameters in accordance with various embodiments of theinvention. Note that the imaging parameters can be dynamicallyreconfigured with any suitable frequency. For example, in manyembodiments, the imaging parameters can be adjusted every 5-10 frames.While the discussion above references adjustment based upon lookuptables, in a number of embodiments determinations of requiredadjustments are made in real time based upon sensor information. Incertain embodiments, optical flow cameras are included in the camerasystem to enable precise measurement of the motion of the main sensorused to perform TDI relative to the scene. As can readily beappreciated, any of a variety of additional sensor systems can beutilized to determine the appropriate rate at which to shift accumulatedintensity information between rows, integration time, and/or the numberof rows over which to accumulate intensity information for each of thedistinct regions of an image sensor used to perform TDI in accordancewith various embodiments of the invention. Accordingly, the specificmanner in which an application processor within a satellite determinesan updated set of imaging parameters to provide to a microcontrollerand/or FPGA that is coordinating TDI and/or controlling readout ofaccumulated intensity information from a CCD image sensor in accordancewith an embodiment of the invention is typically dependent upon therequirements of a given imaging application. In several applications, astrategy is adopted that features frame to frame adjustment of camerasettings (exposure times, analog gains, etc.) to accommodate rapidchanges in altitude or scene brightness.

Although the present invention has been described in certain specificaspects, many additional modifications and variations would be apparentto those skilled in the art. For instance, any of a variety of distinctimaging regions can be implemented in accordance with embodiments of theinvention. It is therefore to be understood that the present inventionmay be practiced otherwise than specifically described. Thus,embodiments of the present invention should be considered in allrespects as illustrative and not restrictive. Furthermore, the scope ofthe invention should be determined not by the embodiments illustratedand described, but by the appended claims and their equivalents.

What is claimed is:
 1. A camera system comprising: an optical system; aCCD image sensor, itself comprising a plurality of pixels that define atleast two distinct imaging regions; where a first distinct imagingregion is configured to image a scene in a single channel; and a seconddistinct imaging region is configured to image a scene in multiplechannels, and where the pixels within each distinct imaging region areconfigured to image a scene differently than at least one other distinctimaging region; a CCD image signal processor; a microprocessor; anamplifier operable to amplify accumulated intensity information read outfrom the CCD image sensor; and an analog to digital converter operableto receive amplified accumulated intensity information, and output adigital representation of the accumulated intensity information; whereinthe optical system is configured to focus incident electromagnetic wavesonto the CCD image sensor; and wherein the camera system is operable ina time-delay integration mode whereby time delay-integration imagingtechniques are imposed such that the camera system operates to read outaccumulated intensity information from each distinct imaging region byrepeatedly: accumulating intensity information in pixels within each rowof pixels within a distinct imaging region; shifting accumulatedintensity information to a subsequent row of pixels within the distinctimaging region, where a last row of pixels does not have a next row ofpixels and instead shifts accumulated intensity information into anarray of accumulators; and reading out accumulated intensity informationfrom the array of accumulators.
 2. The camera system of claim 1, whereinthe second distinct imaging region employs a filter pattern includingcolor filters selected from the group consisting of at least one redcolor filter, at least one green color filter, at least one blue colorfilter, and at least one yellow color filter.
 3. The camera system ofclaim 1, wherein the second distinct imaging region includes a repeatedpattern of filters.
 4. The camera system of claim 3, wherein therepeated pattern of filters repeats a pattern of filters applied topixels in a region of at least 2 pixels×at least 2 pixels.
 5. The camerasystem of claim 3, wherein the repeated pattern of filters repeats apattern of filters applied to pixels in a region of 2 pixels×2 pixels.6. The camera system of claim 5, wherein the pattern of filters appliedto pixels in a region of 2 pixels×2 pixels applies the same filter to atleast two of the pixels and a different filter to at least one of thepixels.
 7. The camera system of claim 6, wherein the pattern of filtersis a Bayer pattern.
 8. The camera system of claim 6, wherein the patternof filters comprises filters that select different wavelengths of redlight.
 9. The camera system of claim 6, wherein the pattern of filterscomprises filters that select different wavelengths of blue light. 10.The camera system of claim 6, wherein the pattern of filters comprisesfilters that select different wavelengths of green light.
 11. The camerasystem of claim 3, wherein the repeated pattern of filters is a repeatedpattern of transmissivity filters.
 12. The camera system of claim 11,wherein the transmissivity filters are dichroic filters.
 13. The camerasystem of claim 12, further comprising a primary filter located betweenthe optical system and the CCD image sensor.
 14. The camera system ofclaim 13, wherein the primary filter is a butcher block filter layercomprising a plurality of regions possessing different filteringcharacteristics.
 15. The camera system of claim 11, wherein the repeatedpattern of transmissivity filters includes filters having differenttransmissivity to enable high dynamic range imaging.
 16. The camerasystem of claim 1, wherein the first distinct imaging region isconfigured to image with a fill factor that is smaller than the fillfactor of the pixels in the second distinct imaging region.
 17. Thecamera system of claim 1, wherein at least one of the distinct imagingregions defines a third distinct imaging region configured to imagenear-infrared electromagnetic radiation.
 18. The camera system of claim1, wherein: the at least two distinct imaging regions include an imagingregion having a repeated pattern of filters that repeats a pattern offilters applied to pixels in a region of at least 2 pixels×at least 2pixels; and shifting accumulated intensity information to a next row ofpixels within the imaging region having the repeated pattern of filterscomprises shifting the accumulated intensity information to anon-adjacent row of pixels.
 19. The camera system of claim 1, whereinthe single and multiple channels represent a sub-portion or sub-portionsof the electromagnetic spectrum.
 20. The camera system of claim 1,wherein at least one of the distinct imaging regions defines a thirddistinct imaging region configured to image ultraviolet electromagneticradiation.
 21. The camera system of claim 1, wherein shiftingaccumulated intensity information to a next row of pixels within atleast one of the distinct imaging regions comprises shifting theaccumulated intensity information to an adjacent row of pixels.
 22. Thecamera system of claim 1, wherein shifting accumulated intensityinformation to a next row of pixels within at least one of the distinctimaging regions comprises shifting the accumulated intensity informationto a non-adjacent row of pixels.
 23. The camera system of claim 1,wherein the camera system is operable to adjust imaging parameters basedupon at least one factor selected from the group consisting of receivedpositioning information, trajectory, and field of view.